Skip to content

Add atomic Content database row migrations - #2573

Open
3mdistal wants to merge 15 commits into
mainfrom
codex/content-atomic-database-migration
Open

Add atomic Content database row migrations#2573
3mdistal wants to merge 15 commits into
mainfrom
codex/content-atomic-database-migration

Conversation

@3mdistal

@3mdistal 3mdistal commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Problem

Content exposes reliable single-row edits, but a bounded reorganization that must update every existing row body and schema cannot safely loop those actions. A timeout or retry could otherwise leave a database partially migrated, with no durable way to distinguish an incomplete write from a completed response that was lost.

The migration also has to coordinate with ordinary Content lifecycle writers. Without one lock order and post-lock revalidation, property/schema changes, trash, restore, and permanent deletion can race the migration or act on a stale deletion scope.

Approach

Add one Content-specific action for whole-database row migrations. It validates the exact active row set, applies one transaction, requires separate readback verification before destructive finalization, and keeps rollback and terminal retries receipt-backed and drift-guarded.

Participating Content lifecycle operations use database → sorted membership lock order. Permanent deletion now rebuilds and revalidates its full database-and-membership scope after acquiring those locks; if a concurrent membership expands the ordered lock set, the entire transaction retries from a fresh snapshot.

This PR deliberately does not add a framework-generic bulk writer, repair MCP authentication, implement a shared document-lifecycle protocol for every create/reparent writer, deploy anything, or migrate any real Content database.

What changed

  • Added migrate-content-database-rows with validate, apply, verify, rollback, and finalize phases.
  • Added additive migration receipts keyed by database and idempotency key, with an in-transaction applying claim before row mutation.
  • Made terminal phase replays observationally inert across rows, schema, versions, receipts, and modification timestamps.
  • Split semantic lock acquisition from explicit modification timestamp updates.
  • Enforced database → sorted membership ordering for migration, property/schema writers, recursive trash, direct database trash, permanent deletion, and workspace recursive deletion.
  • Reloaded and revalidated protected values, schema, timestamps, membership, and deletion scope after locks and live-editor flushes.
  • Made permanent deletion include same-Trash-root rows inserted before the lock is won, fail closed on active rows outside the Trash scope, and retry when a new external membership expands the required lock set.
  • Kept workspace deletion's two canonical roots in one ordered lock scope.
  • Reconciled merged stable-key upserts with the same database lock boundary, retained claim cleanup/duplication guards, and placed migration receipts at additive migration version 80 after stable-key versions 78/79.
  • Preserved the public action schema, receipt format, document and membership identities, ownership, privacy, shares, Status values, and membership-only removal contract.
  • Added synthetic 20-row coverage plus SQLite, PGlite, and real multi-connection PostgreSQL lifecycle/race tests. No private feedback database is used.

Safety and operations

The action accepts ordinary databases without attached Sources and is capped at 100 rows, properties, options, and protected values. Database admin and per-row access checks are required. Content, identity, ownership, personal-space metadata, privacy, shares, property descriptions, source mappings, and values participate in semantic drift checks.

Rollback is available only while the guarded post-apply state still matches. Finalization removes only receipt-recorded legacy properties after the separate verified state. The schema migration is additive. Remote shared SQLite-family backends reject new mutations before any editor flush; existing terminal receipts remain replayable no-ops.

No production or private Content data was read or changed. All test documents, PostgreSQL clusters, PGlite installs, and temporary driver links were synthetic, bounded, cleaned up, and read back as absent.

Verification

Exact head: 9082af762

  • Exact-current focused migration/lifecycle/recursive-delete suites: 58 tests passed.
  • pnpm test:content-db: 13 files and 200 tests passed across the complete migration action suite on merged main.
  • Isolated PGlite harness: lock-lifetime probe plus 14 applicable action tests passed; one SQLite-trigger-only rollback test is intentionally skipped on PGlite.
  • Real PostgreSQL 17: 13 tests passed across multiple connections, including apply/rollback editor flushes, property/schema races, Trash-scope expansion, external-membership retry, restore-vs-delete, and receipt cleanup.
  • Exact-current pnpm --filter content typecheck and pnpm fmt:check: passed.
  • pnpm test:content-product-impact: 30 tests passed.
  • pnpm guards: all 42 checks passed on the sealed merge commit.
  • pnpm oxlint: passed with repository baseline warnings only.
  • Human QA skipped: this is a headless action/transaction contract with no changed interactive UI; backend action entrypoints and real database concurrency are the acceptance surface.
  • Exact-head GitHub CI passed, including Content DB, real PostgreSQL locking, parity, build, typecheck, security, and every fast-test lane.
  • Builder's Review Agent reported skipped on this merge head and did not publish a re-runnable check. Fresh independent technical review therefore remains pending; the older approval is not represented here as exact-head review.

Review focus

  • Are terminal receipt replays truly read-only across every durable table and timestamp?
  • Does the backend split serialize editor saves without deadlocking SQLite/PGlite or weakening PostgreSQL cross-process safety?
  • Does permanent deletion stabilize the database-and-membership scope before deleting any document or receipt?
  • Are active out-of-scope database rows rejected without leaving a partial deletion?
  • Is the separate shared document-lifecycle protocol correctly excluded from this migration-safety PR?

Follow-up

S2573-F1, the shared document-lifecycle protocol across create, membership, and reparent writers, remains a separate shaped workstream. Its preserved experiment is not part of this branch and stays on hold pending the Content implementation-versus-roadmap audit.

content_product_impact:
  lane: contract_fulfillment
  features:
    - content.feature.durable-foundations
  capabilities:
    - content.object.database
    - content.agent.action-parity
  record_change: none
  proof:
    - pnpm test:content-db
    - pnpm test:content-db-postgres
    - pnpm test:content-parity
  rationale: Adds one bounded, receipt-backed Content action so agents can safely reorganize an existing database without partial row writes or replaced identities.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Visual recap — screenshot failed

A recap was published, but the PR-comment screenshot could not be captured or uploaded. Open the interactive recap directly:

Open the full interactive recap

Diagnostic:

light: page.screenshot: Timeout 30000ms exceeded. Call log: - taking page screenshot - waiting for fonts to load...

builder-io-integration[bot]

This comment was marked as outdated.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

@netlify

This comment has been minimized.

builder-io-integration[bot]

This comment was marked as outdated.

…database-migration

# Conflicts:
#	templates/content/actions/delete-database-items.ts
#	templates/content/parity/matrix.md
#	templates/content/parity/matrix.ts
…database-migration

# Conflicts:
#	package.json
builder-io-integration[bot]

This comment was marked as outdated.

builder-io-integration[bot]

This comment was marked as outdated.

@netlify

This comment has been minimized.

builder-io-integration[bot]

This comment was marked as outdated.

builder-io-integration[bot]

This comment was marked as outdated.

@builder-io-integration builder-io-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Builder reviewed your changes and found 2 potential issues 🔴

Review Details

Incremental Code Review Summary

The existing restore-versus-migration issue remains unresolved and was not reposted. The latest changes add broader trash/permanent-delete lock helpers and extensive lifecycle race coverage, but the permanent-delete path still has pre-lock collection windows that can invalidate the lock set and deletion inputs.

This remains high risk because recursive deletion mutates database memberships, documents, and receipt-bearing state across multiple databases. Two new high-severity findings were identified:

  • 🔴 HIGH — Permanent deletion can discover external memberships before locking, then fail to lock a database attached during the gap.
  • 🔴 HIGH — Permanent deletion can use a stale pre-lock subtree, delete a newly added membership/database record, and leave the newly created row document orphaned.

🧪 Browser testing: Skipped — PR changes backend actions, persistence, tests, CI, parity metadata, and documentation; no user-facing UI implementation changed.

Comment thread templates/content/actions/delete-document.ts Outdated
Comment thread templates/content/actions/delete-document.ts Outdated
@netlify

This comment has been minimized.

@builder-io-integration builder-io-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Builder reviewed your changes and found 1 potential issue 🔴

Review Details

Incremental Code Review Summary

The prior restore-versus-migration issue is fixed: restore now collects its scope, acquires database and membership locks, rechecks scope coverage, and only then restores rows. That thread was resolved. The two prior permanent-deletion lock/rebuild comments remain open and were not reposted.

The latest lifecycle changes are otherwise well covered by SQLite/PGlite/PostgreSQL tests. One new high-severity race remains in permanent deletion: a trashed subtree is collected before waiting for its locks, and the Trash state is not revalidated after the locks are acquired.

New finding

  • 🔴 HIGH — A concurrent restore can complete while permanent deletion waits for the database lock; deletion then proceeds with the stale document set and deletes the restored subtree because cleanup lacks a Trash-state predicate.

🧪 Browser testing: Skipped — PR changes backend actions, persistence, tests, CI, parity metadata, and documentation; no user-facing UI implementation changed.

Comment thread templates/content/actions/delete-document.ts Outdated
…database-migration

# Conflicts:
#	templates/content/actions/configure-document-property.ts
#	templates/content/actions/delete-content-database.ts
#	templates/content/actions/delete-document-property.ts
#	templates/content/actions/delete-document.ts
#	templates/content/actions/duplicate-database-item.ts
#	templates/content/actions/duplicate-database-items.ts
#	templates/content/actions/permanently-delete-document.ts
#	templates/content/actions/set-document-property.ts
#	templates/content/parity/matrix.md
#	templates/content/server/plugins/db.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants